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Large numbers of U.S. students lack proficiency in science, 
and students from different racial/ethnic and socioeconomic 
groups show disparities in science achievement. Science 
knowledge and skills are important for both academic and 
workplace success, and a variety of interventions have been 
developed to improve student achievement in science. 
These interventions include professional development 
programs for teachers that are designed to change teachers’ 
content knowledge or classroom instruction in science. 
Science Teachers Learning through Lesson Analysis (STeLLA”) 
is a professional development program, developed by BSCS 
Science Learning, that aims to improve students’ science 
achievement by improving teachers’ science content 
knowledge and their abilities to (a) explain science concepts 
to students, (b) clearly identify to students the science 
concepts used in student learning activities, and (c) engage 
students in thinking about science. 


Participating teachers learn from postsecondary science faculty 
to improve their knowledge of the science content areas they 
will teach during the school year; become familiar with STeLLA® 
lesson plans and instructional strategies; work in study groups 
led by STeLLA’® professional development leaders to analyze 
videorecorded examples of experienced teachers implementing 
STeLLA* lesson plans and strategies; and videorecord their 
own classroom instruction and meet with their study group 
to analyze one another’s videorecorded lessons. 


This What Works Clearinghouse (WWC) intervention report, part 
of the WWC’s Primary Science topic area, explores the effects 
of STeLLA® on science achievement. The WWC identified three 
studies of STeLLA’. One of these studies meets WWC standards. 
The evidence presented in this report is from one study of the 
effects of STeLLA® on students—including 59% White, 4% Black, 
2% Asian, and 28% Hispanic students—in grades 4 and 5 in 
urban, suburban, and rural schools in Colorado. 


What Happens When Students Participate in STeLLA*? 


The evidence indicates that implementing STeLLA’® has 
potentially positive effects on science achievement 
compared with another yearlong teacher professional 
development program. 


Findings on STeLLA® from the one study that meets WWC 
standards are shown in Table 1. The table reports an 
effectiveness rating, an improvement index, and the number 
of studies and students that contributed to the findings. The 
effectiveness rating is based on the quality of the designs 
used in studies, whether the findings are favorable or 
unfavorable for the intervention, and the number of studies 
that tested the intervention. See Box 1 for more information 
on interpreting effectiveness ratings. 


In order to help readers judge the practical importance of 
an intervention’s effect, the WWC translates findings across 
studies into an “improvement index” by averaging findings 


that meet WWC standards within the same outcome domain. 
The improvement index can be interpreted as the expected 
change in percentile rank for an average comparison group 
student if that student had received the intervention. For 
example, an improvement index of +25 means that the 
expected percentile rank of the average comparison group 
student would increase by 25 points if the student received 
instruction from a teacher who had participated in STeLLA® 
instead of the professional development program provided 
to the comparison group teachers. A positive improvement 
index does not necessarily mean the estimated effect is 
statistically significant. Results for each individual outcome 
measure within domains are shown in Table 4. 


The evidence presented in this report is based on available 
research. Findings and conclusions could change as new 
research becomes available. 


Table 1. Summary of findings on STeLLA’ from one study that meets WWC standards 


Study Findings 


Evidence meeting WWC standards (version 4.0) 


Improvement index 


Outcome domain Effectiveness rating (percentile points) Number of studies Number of students 


Science achievement Potentially positive effects +25 1 2,823 


Note: For more information about outcome measures, see the study description in Table 6. The effects of STeLLA® are not known for other outcomes within the Primary Science 
topic area, including life sciences, physical sciences, and earth/space sciences. 


BOX 1. HOW THE WWC REVIEWS AND DESCRIBES EVIDENCE 


The WWC evaluates evidence based on the quality and results of reviewed studies. The criteria the WWC uses for evaluating 
evidence are defined in the Procedures and Standards Handbooks and the Review Protocols. The studies summarized in this report 
were reviewed under WWC Standards (version 4.0) and the Primary Science topic area protocol (version 4.0). 


To determine the effectiveness rating, the WWC considers what methods each study used, the direction of the effects, and the 
number of studies that tested the intervention. The higher the effectiveness rating, the more certain the WWC is about the reported 
results and about what will happen if the same intervention is implemented again. The following key explains the relationship between 
effectiveness ratings and the statements used in this report: 


Effectiveness Rating 


Rating interpretation 


Description of the evidence 


Positive (or negative) effects 
outcome 


The intervention is likely to change an 


Strong evidence of a positive (or negative) 
effect, with no overriding contrary evidence 


Potentially positive (or negative) effects 


The intervention may change an outcome 


Evidence of a positive (or negative) effect with 
no overriding contrary evidence 


No discernible effects 


The intervention may result in little to no 
change in an outcome 


No affirmative evidence of effects 


Mixed effects 
on an outcome 


How is STeLLA’ Implemented? 


The following section provides details of how districts and 
schools can implement STeLLA’. This information can help 
educators identify the requirements for implementing 
STeLLA® and determine whether implementing this 
intervention would be feasible in their districts or schools. 
Information on STeLLA” presented in this section comes 
from the study that meets WWC standards (Taylor et al., 
2017) and from correspondence with the developer. 


¢ Goal: STeLLA® aims to improve student science achieve- 
ment by improving a teacher’s ability to (a) explain sci- 
ence concepts to students, (b) clearly identify to students 
the science concepts used in student learning activities, 
and (c) engage students in thinking about science. 


¢ Target population: STeLLA’ is offered to teachers who 
provide science instruction to students in kindergarten 
to grade 12. Training is also available for school or district 
personnel who wish to become STeLLA* professional 
development leaders. 


¢ Method of delivery: Teachers learn from postsecondary 
science faculty and STeLLA® professional development 
leaders at an in-person summer institute. During the 
institute, teachers begin improving their science content 
knowledge, become familiar with STeLLA® lesson plans 
and instructional strategies, and learn to analyze videos 
of experienced teachers delivering model lesson plans 
and using the instructional strategies. During the school 
year, teachers videorecord themselves delivering model 
STeLLA’ lesson plans and meet in study groups at their 
school with in-person or online guidance from their 
STeLLA* professional development leader to analyze their 
own and one another’s videorecorded lessons. 


The intervention has inconsistent effects 


Evidence includes studies in at least two of 
these categories: studies with positive effects, 
studies with negative effects, or more studies 
with indeterminate effects than with positive or 
negative effects 


Comparison condition: In the one study that con- 
tributes to this intervention report, students in the 
comparison group received science instruction from 
teachers who participated in a yearlong professional 


development program designed to improve only their 
science content knowledge. These teachers did not 
receive STeLLA’® materials or training and support from 
STeLLA’ professional development leaders. 


Frequency and duration of service: STeLLA’ is a year- 
long professional development program for teachers that 
begins with a 2-week summer institute (60 hours) fol- 
lowed by eight monthly grade-level study group meetings 
during the school year (30 hours). Personnel who wish 
to become STeLLA® professional development leaders 
require an additional 1 to 2 years of training. Refer to 
Table 2 for additional details. 


Intervention components: The key components of 
STeLLA® are described in Table 2. Training for personnel 
to become STeLLA® professional development leaders, 
which can be bundled with the teacher program, is also 
described in Table 2 but was not part of the study that 
contributes to this intervention report. 


Table 2. Components of STeLLA® 


Key component 


Description 


Teacher professional 
development 


STeLLA® lesson plans 


STeLLA® teaching 
strategies 


Analysis of 
videorecorded lessons 
in study groups 


STeLLA® leadership 
development training 


During the yearlong STeLLA® professional development program, teachers (a) develop science content knowledge of two 
topics (for example, “the earth’s changing surface” and “matter and molecules in the water cycle”) that they will teach in 
their own classrooms in the upcoming school year, (b) review and implement STeLLA® model lesson plans, and (c) learn 
instructional strategies to support student learning. 


The professional development program begins with a 2-week (58 to 60 hours) summer institute facilitated by postsecondary 
science faculty and a STeLLA® professional development leader. Science faculty engage teachers in short lectures, readings, 
activities, and discussions of science concepts. STeLLA® professional development leaders lead teachers through a review of 
content-specific background documents that describe how science lesson activities, representations of science concepts, and 
analogies that are typically used in science instruction might support or interfere with student learning. STeLLA® professional 
development leaders also introduce teachers to the STeLLA® lesson plans, teaching strategies, and process for analyzing 
videorecorded lessons. 


During the school year, teachers meet with school-based study groups of other teachers of students in the same grade and their 
STeLLA® professional development leader monthly for eight discussion sessions each lasting 3 to 4 hours to continue learning 
how to convey science content to students and how to implement STeLLA® teaching strategies in their own classrooms. 


At the summer institute, participating teachers receive STeLLA® model lesson plans for one of the two topics. The model 
lesson plans are detailed guides designed to build teachers’ science content knowledge and familiarity with STeLLA® teaching 
strategies. They highlight common student responses to teacher questions and suggest ways for teachers to respond to 
students’ ideas, predictions, and questions. Teachers review these lesson plans and analyze videos of experienced teachers 
delivering them. In the fall, teachers practice implementing the model lesson plans in their own classrooms. 


Teachers learn instructional strategies that emphasize: 


* Explaining science concepts, including using accurate descriptions, analogies, visual displays or models, and other 
representations of scientific content that minimize student misconceptions; allowing students to practice using these 
representations; describing the connections between science concepts; and summarizing key elements of these concepts. 


* Clearly identifying to students the science concepts used in student learning activities, including establishing a 
learning goal with a central question or objective, selecting activities matched to the objective, sequencing science content 
and learning activities appropriately, and identifying how the learning activities illustrate or draw on science concepts. 


¢ Engaging students in thinking about science, including asking questions to elicit and challenge student ideas and 
predictions; engaging students in interpreting data, summarizing science concepts, and applying those concepts to new 
contexts or phenomena; and encouraging students to communicate in scientific ways. 


Teachers first become familiar with STeLLA® strategies at the summer institute by watching videos of experienced teachers 
using them in science lessons. Teachers later practice using the strategies in planning and delivering science instruction in 
their own classrooms. 


A STeLLA?® professional development leader works with school-based study groups to analyze videorecorded examples 
of science instruction and accompanying examples of student written work. Teachers discuss their own and students’ 
understanding of the science topics covered in the lessons and learn to identify STeLLA® instructional strategies that the 
videorecorded teacher used or could have used to support student learning. 


At the summer institute, teachers begin learning the lesson analysis process by discussing videorecorded lessons of 
experienced teachers implementing STeLLA® model lesson plans. During the school year, teachers videorecord themselves 
teaching the two science topics in their own classrooms and then analyze one another’s videorecorded lessons and 
accompanying examples of student work. In the first half of the school year, teachers analyze videos of one another delivering 
STeLLA® model lesson plans. In the second half of the school year, teachers collaborate with their study group to develop 
their own science lesson plans for the second science topic by incorporating STeLLA® teaching strategies and anticipating 
student misconceptions. Teachers then deliver and videorecord these lessons and meet once more to analyze one another's 
instruction and summarize what they have learned over the yearlong program. 


Also available is a 2- to 3-year program for school or district personnel to become STeLLA® professional development leaders. 
Participants complete 1 year of leadership training before they begin providing STeLLA® professional development to a cohort 
of teachers. During the second year of the program, the leaders-in-training receive support from the STeLLA® developer. This 
component was not part of the study that contributes to this intervention report. 


What Does STeLLA® Cost? 


This preliminary list of costs is not designed to be exhaustive; 
rather, it provides educators an overview of the major 
resources needed to implement STeLLA’. The program costs 


Table 3. Cost ingredients for STeLLA° 


described in Table 3 are based on the information available 
as of May 2020. 


Cost ingredients Description 


Source of funding 


Implementing STeLLA® costs approximately $1,750 per teacher for the 
yearlong program. Each participating teacher attends a 2-week summer 
institute and eight monthly study group sessions (up to 4 hours each) during 
the school year. A 2-year leadership development program for school or 
district personnel to become STeLLA® professional development leaders 
costs approximately $10,000 per leader, but this component is not required. 


Schools and districts pay BSCS Science Learning 
for costs of providing the STeLLA® summer institutes 
and support for participating teachers (or leaders-in- 
training) throughout the following school year. 


The 2-week summer institute typically takes place at a facility provided by the 
school district. School-year study group meetings take place at participating 
schools, with the professional development leader joining online or on-site 
depending on the availability of local STeLLA® professional development 
leaders and the agreement with BSCS Science Learning. 


Schools and districts are responsible for providing 
the facility for the summer institute and the transportation 
to and from the facility. Schools and districts provide 
physical space for teacher professional development 
and classroom instruction during the school year. 


Teacher participants receive a set of grade-appropriate STeLLA® lesson plans 
for one of the two topics they will teach in the upcoming year, transcripts of 
program-developed videorecorded lessons and related examples of student 
work, a STeLLA® strategy guide, lesson analysis protocols, and other resources 
associated with STeLLA®. BSCS Science Learning provides teachers with 
online access to videorecordings made by members of their local study group. 


Participants training to become STeLLA® professional development leaders receive 
the same materials provided to teacher participants as well as videorecordings 
and transcripts from STeLLA® professional development sessions with teachers 


The fees that schools or districts pay for teacher 

(or leader-in-training) professional development 
cover the cost of program materials provided to 
participants. Schools and districts are responsible for 
the costs of videorecording participating teachers. 


and additional resources specific to the leadership development program. 


For More Information: 
About STeLLA® 


BSCS Science Learning, 5415 Mark Dabling Boulevard, Colorado Springs, CO 80918 
Attn: Jody Bintz, Associate Director for Strategic Partnerships & Professional Learning 


Email: jbintz2@bscs.org Web: https://oscs.org/our-work/rd-programs/stella-science-teachers-learning-from-lesson-analysis/. 


Phone: (719)-513-5550 
About the cost of the intervention 


Actual program costs available in consultation with the developer. 


Research Summary 


The WWC identified three studies that investigated the 
effectiveness of STeLLA® (Figure 1): 


¢ 1study meets WWC group design standards without 
reservations 


¢ 2 studies do not meet WWC group design standards 


The WWC reviews findings on the intervention’s effects 
on eligible outcome domains from studies that meet 
standards, either with or without reservations. Based on this 


review, the WWC generates an effectiveness rating, which 
summarizes how the intervention impacts, or changes, a 
particular outcome domain. 


The one study of STeLLA® that meets WWC group design 
standards reported findings on science achievement. No other 
findings in the study meet WWC group design standards within 
any outcome domain included in the Primary Science topic 
area.' Citations for the three studies reviewed for this report 
are listed in the References section, which begins on page 8. 


Figure 1. Effectiveness ratings for STeLLA® 


study meets WWC Gade meet WWC 


standards with 
reservations 


standards without 
reservations 


Contribute to effectiveness ratings 


studies do not 0) studies are 


meet WWC 
standards 


ineligible for 
review 


Do not contribute to effectiveness ratings 


STeLLA® has potentially positive effects on science achievement 


The WWC determined that one study that meets WWC group design standards without reservations shows evidence 
of a positive and statistically significant effect of STeLLA® on science achievement (Taylor et al., 2017). 


Main Findings 


Table 4 shows the findings from the one STeLLA’® study that 
meets WWC standards. The table includes WWC calculations 
of the performance of the intervention group relative to 

the comparison group in terms of the mean difference and 
effect size. The effect size is a standardized measure of the 
effect of an intervention on outcomes, representing the 
average change expected for all individuals who are given 
the intervention (measured in standard deviations of the 
outcome measure). For the mean difference and effect size 


values, a positive number favors the intervention group and 
a negative number favors the comparison group. A positive 
or negative improvement index does not necessarily mean 
the estimated effect is statistically significant. 


Based on findings from the one study that meets WWC 
standards and includes 2,823 students, the effectiveness 
rating for science achievement is potentially positive effects. 


Table 4. Findings by outcome domain from study of STeLLA’ that meets WWC standards 


Mean 
(standard deviation) WWC calculations 
Sample Intervention Comparison Mean Effect Improvement 

Measure (study) Study sample size group group difference size index p-value 
Researcher-developed Students in 2,823 56.28 50.17 6.11 0.68 +25 .00 
science content grades 4 and 5 (9.88) (7.71) 
knowledge test 
(Taylor et al., 2017)# 

: ; Statistically 
Outcome average for science achievement (Taylor et al., 2017) 0.68 +25 significant 


Notes: Some statistics may not sum as expected due to rounding. 


4 Taylor et al. (2017) developed a science content knowledge test for this study to measure students’ knowledge of science content in four areas: the earth's changing surface, 
energy and matter in food webs, the sun’s effect on climate and seasons, and matter and molecules in the water cycle. The study is characterized as having a statistically 
significant positive effect on science achievement because the estimated effect is positive and statistically significant. For more information, please refer to the WWC. 


Procedures Handbook, version 4.0, page 22. 


In What Context Was STeLLA® Studied? 


The following section provides information on the setting of 
the one study of STeLLA’ that meets WWC standards, and a 
description of the participants in the research. This information 


WHERE THE STUDY WAS CONDUCTED 


ee 1 study, 2,823 students in 77 elementary schools in Colorado 


Race 4% 2% 


Ce) 


White 


Free & Reduced-Price Lunch: 43% 


Grades 


Black Asian Other/Not specified 


English Learners: 12% 


can help educators understand the context in which the 
study of STeLLA® was conducted and determine whether the 
program might be suitable for their setting. 


Urban, suburban, and rural districts 


Ethnicity 


Hispanic Non-Hispanic 


Gender: 48% Female 52% Male 


Details of Each Study that Meets WWC Standards 


This section presents details for the one study of STeLLA® 
that meets WWC standards. These details include the full 
study reference, findings description, findings summary, 
and description of study characteristics. A summary of 
domain findings for the study is presented below, followed 
by a description of the study characteristics. These study- 
level details include contextual information about the study 
setting, methods, sample, intervention group, comparison 
group, outcomes, and implementation details. For additional 
information, readers should refer to the original study. 


Research details for Taylor et al. (2017) 


Taylor, J., Roth, K., Wilson, C. Stuhlsatz, M., & Tipton, E. (2017). 
The effect of an analysis-of-practice, videocase-based, teacher 
professional development program on elementary students' 
science achievement. Journal of Research on Educational 
Effectiveness, 10(2), 241-271. https://eric.ed.gov/?id=EJ1135795 


Findings from Taylor et al. (2017) show evidence of a statistically 
significant positive effect of STeLLA’ on science achievement 
(Table 5). The findings and research details summarized 

for this study come from two related citations, including 

the primary study listed above. See the References section, 
which begins on page 8, for a list of all related publications. 


Table 5. Summary of findings from Taylor et al. (2017) 


Meets WWC Group Design Standards Without Reservations 


Study findings 
P Sample Average Improvement Statistically 
Outcome domain size effect size index significant 
Science achievement 77 schools, 2,823 students 0.68 +25 Yes 


Table 6. Description of study characteristics for Taylor et al. (2017) 


WWC evidence 
rating 


Setting 


Methods 


Study sample 


Intervention 
condition 


Comparison 
condition 


Outcomes and 
measurement 


Additional 
implementation 
details 


Meets WWC Group Design Standards Without Reservations. This is a cluster randomized controlled trial with low cluster- 
level attrition and low individual-level nonresponse. For more information on how the WWC assigns study ratings, please see 
the WWC Procedures and Standards Handbooks (version 4.0) and WWC Standards Briefs, available on the WWC website. 


The study took place during science instruction in fourth- and fifth-grade classrooms in 77 public elementary schools in 
urban, suburban, and rural areas of Colorado. The sample consisted of traditional public schools and excluded charter and 
magnet schools. The study authors did not specify the number of districts. 


The study assigned schools to implement one of two professional development programs, STeLLA® (the intervention 
condition) or a program focused on improving teachers’ understanding of science content (the comparison condition). Two 
cohorts of schools participated for 1 year each, one in 2011-12 and another in 2012-13, with random assignment of each 
cohort occurring in the summer before the start of each school year. Study authors combined results across the two cohorts. 


Across the two cohorts, researchers randomly assigned a total of 84 schools (46 in the intervention group and 38 in the 
control group) with 3,710 students (1,979 students in the intervention group schools and 1,731 students in the control group 
schools). The sample loss after random assignment (attrition) was within the acceptable threshold for the review. At the 
school level, the overall attrition rate was 8.3% and the differential attrition rate was less than 1 percentage point. At the 
student level, the overall nonresponse rate was 24% and the differential nonresponse rate was 2.3 percentage points. 


The study sample included 2,823 students (1,485 in the intervention group and 1,338 in the comparison group) in fourth- 
and fifth-grade classrooms in the participating schools. In the participating schools, 52% of students were male, 59% of the 
students were White, 4% were Black, 2% were Asian, and 28% were Hispanic, on average. Twelve percent of students in 
the participating schools had limited English proficiency, and 43% qualified for free or reduced-price lunch, on average. 


Over the course of 1 school year, teachers in the intervention group participated in the STeLLA® professional development 
program, starting with a 2-week summer institute (approximately 60 hours), followed by eight once-monthly meetings 
during the school year in school-based study groups (approximately 30 hours). Teachers focused on two science topics 
that they would teach in their own classrooms in the upcoming school year: “the earth's changing surface” and “food webs” 
for fourth-grade teachers, and “matter and molecules in the water cycle” and “the sun’s effect on climate/seasons’ for fifth- 
grade teachers. In the summer institute, teachers alternated half-day sessions on developing science content knowledge, 
led by postsecondary science faculty, with half-day sessions led by STeLLA® professional development staff in which 

they became familiar with STeLLA® lesson plans and teaching strategies and analyzed videorecordings of experienced 
teachers delivering model lesson plans. During the school year, teachers videorecorded themselves delivering lesson plans 
(program-provided or developed with their study group) and participated in monthly 3- to 4-hour school-based study group 
sessions facilitated by a STeLLA® professional development leader to analyze one another's instruction. 


Teachers in the comparison group participated in a 1-year professional development program focused on developing 
science content knowledge in the same two topic areas as the intervention group: “the earth’s changing surface” and “food 
webs” for fourth-grade teachers and “matter and molecules in the water cycle” and “the sun’s effect on climate/seasons’” for 
fifth-grade teachers. The program began with a 2-week summer institute (approximately 60 hours), followed by five 6-hour 
meetings during the school year (30 hours total). Postsecondary science faculty led both the summer institute and school- 
year sessions. In contrast to teachers in the intervention group, teachers in the comparison group did not receive training or 
support from STeLLA® professional development leaders. Teachers in the comparison group also did not receive STeLLA® 
lesson plans, learn about instructional strategies, nor participate in analysis of videorecorded science instruction. 


Study authors reported findings on a science content knowledge test within the science achievement domain that they 
developed using a combination of new items and items from extant tests including the National Assessment of Educational 
Progress, Trends in International Mathematics and Science Study, and assessments developed by other researchers 

and funded by National Science Foundation grants (see Taylor et al., 2017, pp. 256, 266). The researcher-developed test 
included multiple-choice items covering four topics (the earth’s changing surface, energy and matter in food webs, the sun's 
effect on climate and seasons, and matter and molecules in the water cycle). Study findings were based on students’ spring 
scores, which covered two of the four topics. The test was administered 2 weeks after the teacher completed instruction on 
the topics. The study authors used statistical methods to generate a single measure of science content knowledge for each 
student. This researcher-developed measure had a reliability of 0.83, which meets the WWC’'s outcome requirements.’ 


An external evaluator conducted an implementation study and gave similar ratings to the professional development provided 
to the intervention and comparison groups. The ratings were for the following indicators: effectiveness of the professional 
development provider, pacing of the sessions, teacher engagement, and collaboration among teachers. Observers rated the 
STeLLA® intervention group as spending more time on pedagogical issues and lesson analysis, and less time on science 
content, than the comparison group. 
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Endnotes 


'The effects of STeLLA’ are not known for other outcome 
domains within the Primary Science topic area, including 
life sciences, physical sciences, and earth/space sciences. 


?The authors used Rasch common person equating to esti- 
mate each student’s science content knowledge score on 
an equal interval scale and reported the measure’s Rasch 
person reliability. The authors compared Rasch common 
person equating to item-equating processes used in tests 
like the National Assessment of Educational Progress, in 
which different students respond to different test items 
but all students receive an estimated score on a common 
scale. See works cited in Taylor et al. (2017): Bond, T. G., 
& Fox, C. M. (2007). Applying the Rasch model fundamental 
measurement in the human sciences (2nd ed.). Lawrence 
Erlbaum; and Boone, W. J., & Scantelbury, K. (2006). The 
role of Rasch analysis when conducting science education 
research utilizing multiple-choice tests. Science Education, 
90(2), 551-569. 
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